• TensorFlow Basics

• R Interfaces to TensorFlow

• Deep Learning

• Supporting Tools

• Deployment

• Learning More

Rapid development, many outside contributors

• 1,200+ contributors to TensorFlow (950+ non-Google)
• 26,500+ commits
• 41,000+ GitHub repositories (forks)
• 9.8M+ downloads
• 84,000+ stars on Github
• ~8,000 GitHub issues filed and answered
• 10,000 Stack Overflow questions answered

Use in ML classes is growing: 9 out of top 14 online deep learning courses use TensorFlow

• A new general purpose numerical computing library!
  • Hardware independent
  • Distributed execution
  • Large datasets
  • Automatic differentiation

• Robust foundation for machine learning and deep learning applications

• TensorFlow models can be deployed with a low-latency C++ runtime

• R has a lot to offer as an interface language for TensorFlow

https://greta-dev.github.io/greta/

# Greta
theta = normal(0, 32, dim = 2)
mu <- alpha + beta * Z
X = normal(mu, sigma)
p <- ilogit(theta[1] + theta[2] * X)
distribution(y) = binomial(n, p)

# BUGS/JAGS
for(j in 1 : J) {
   y[j] ~ dbin(p[j], n[j])
   logit(p[j]) <- theta[1] + theta[2] * X[j]
   X[j] ~ dnorm(mu[j], tau)
   mu[j] <- alpha + beta * Z[j]
}
theta[1] ~ dnorm(0.0, 0.001)
theta[2] ~ dnorm(0.0, 0.001)

• Vector data—2D tensors of shape (samples, features)

• Timeseries or sequence data—3D tensors of shape (samples, timesteps, features)

• Images—4D tensors of shape (samples, height, width, channels) 

• Video—5D tensors of shape (samples, frames, height, width, channels)

head(data.matrix(iris), n = 10)

      Sepal.Length Sepal.Width Petal.Length Petal.Width Species
 [1,]          5.1         3.5          1.4         0.2       1
 [2,]          4.9         3.0          1.4         0.2       1
 [3,]          4.7         3.2          1.3         0.2       1
 [4,]          4.6         3.1          1.5         0.2       1
 [5,]          5.0         3.6          1.4         0.2       1
 [6,]          5.4         3.9          1.7         0.4       1
 [7,]          4.6         3.4          1.4         0.3       1
 [8,]          5.0         3.4          1.5         0.2       1
 [9,]          4.4         2.9          1.4         0.2       1
[10,]          4.9         3.1          1.5         0.1       1

• High-level R interfaces for neural nets and traditional models

• Low-level interface to allow enable new applications (e.g. Greta)

• Tools to faciliate productive workflow / experiment management

• Easy access to GPUs for training models

• Breadth and depth of educational resources

• High-level neural networks API capable of running on top of TensorFlow, CNTK, or Theano (and soon MXNet).

• Allows for easy and fast prototyping (through user friendliness, modularity, and extensibility).

• Supports both convolutional networks and recurrent networks, as well as combinations of the two.

• Runs seamlessly on CPU and GPU.

• https://keras.rstudio.com

model <- keras_model_sequential() %>%
  layer_conv_2d(filters = 32, kernel_size = c(3,3), activation = 'relu',
                input_shape = input_shape) %>% 
  layer_conv_2d(filters = 64, kernel_size = c(3,3), activation = 'relu') %>% 
  layer_max_pooling_2d(pool_size = c(2, 2)) %>% 
  layer_dropout(rate = 0.25) %>% 
  layer_flatten() %>% 
  layer_dense(units = 128, activation = 'relu') %>% 
  layer_dropout(rate = 0.5) %>% 
  layer_dense(units = 10, activation = 'softmax')

library(keras)

# Load MNIST images datasets (built in to Keras)
c(c(x_train, y_train), c(x_test, y_test)) %<-% dataset_mnist()

# Flatten images and transform RGB values into [0,1] range 
x_train <- array_reshape(x_train, c(nrow(x_train), 784))
x_test <- array_reshape(x_test, c(nrow(x_test), 784))
x_train <- x_train / 255
x_test <- x_test / 255

# Convert class vectors to binary class matrices
y_train <- to_categorical(y_train, 10)
y_test <- to_categorical(y_test, 10)

model <- keras_model_sequential()  %>% 
  layer_dense(units = 256, activation = 'relu', input_shape = c(784)) %>% 
  layer_dropout(rate = 0.4) %>% 
  layer_dense(units = 128, activation = 'relu') %>%
  layer_dropout(rate = 0.3) %>%
  layer_dense(units = 10, activation = 'softmax')

model %>% compile(
  loss = 'categorical_crossentropy',
  optimizer = optimizer_rmsprop(),
  metrics = c('accuracy')
)

history <- model %>% fit(
  x_train, y_train,
  batch_size = 128,
  epochs = 30,
  validation_split = 0.2
)

history

Trained on 48,000 samples, validated on 12,000 samples (batch_size=128, epochs=30)
Final epoch (plot to see history):
     acc: 0.9057
    loss: 1.5
 val_acc: 0.9317
val_loss: 1.088 

plot(history)

model %>% evaluate(x_test, y_test)

$loss
[1] 0.1078904

$acc
[1] 0.9815

model %>% predict_classes(x_test[1:100,])

  [1] 7 2 1 0 4 1 4 9 5 9 0 6 9 0 1 5 9 7 3 4 9 6 6 5 4 0 7 4 0 1 3 1 3 4 7
 [36] 2 7 1 2 1 1 7 4 2 3 5 1 2 4 4 6 3 5 5 6 0 4 1 9 5 7 8 9 3 7 4 6 4 3 0
 [71] 7 0 2 9 1 7 3 2 9 7 7 6 2 7 8 4 7 3 6 1 3 6 9 3 1 4 1 7 6 9

• Deep learning was a principle motivator for the creation of TensorFlow

• What is deep learning and what is it useful for?

• How is deep learning relevant to the R community?

Special thanks to François Chollet (creator of Keras) for the concepts and figures used to explain deep learning! (all drawn from Chapter 1 of his Deep Learning with R book)

• Input data points (e.g. images)

• Labels (e.g. cat vs. dog)

• Model architecture (e.g. logistic regression)

• Loss function (measures performance of model, e.g. categorical crossentroopy)

• Loss is used as a feedback signal to an optimizer (e.g. stochastic gradient descent) to iteratively adjust the model parameters

• Training consists of feeding batches of data to training process to learn the optimal model parameters.

Machine-learning algorithms aren’t usually creative in finding these transformations; they’re merely searching through a predefined set of operations, called a hypothesis space.

A new take on learning representations from data that puts an emphasis on learning successive layers of increasingly meaningful representations.

Other possibly more appropriate names for the field:

• Layered representations learning

• Hierarchical representations learning

Modern deep learning often involves tens or even hundreds of successive layers of representation

Other approaches to machine learning tend to focus on learning only one or two layers of representations of the data

Layers perform transformations using weights, but how do we learn the weights?

• Hardware (NVIDIA GPUs)

• Data (internet enabled collection and distribution of very large datasets)

• Various algorithmic advances enabled training models with 10 or more layers

• Research now proceeding at a furious pace!

• Models cannot be easily interpreted (black box)
• Models can be brittle (see "adversarial examples")
• Models are often very expensive to train
• Current generation of tools make deep learning accessible to many with no prior experience in statistics or modeling, however…
• It can be quite difficuilt (or impossible!) to out-perform traditional modeling techniques on many problems.

• New problem domains for R:
  • Computer vision
  • Computer speech recognition and translation
  • Various other forms of complex pattern recognition
• Improved techniques for our traditional domains:
  • Analyzing data with complex spatial or sequence dependencies
  • Analyzing data which requires a large amount of (potentially brittle) feature engineering to model effectively

• Visual recognition

• Language translation

• Time series

• Biology

• What next?

• Image classification on small datasets
• Time series forecasting with recurrent networks
• Peptide classification using convolutional networks
• Credit card fraud detection using an autoencoder
• Classifying duplicate questions from Quora
• Learning word embeddings for Amazon reviews
• Generative models

# use a pretrained model as the base of a new classifier
conv_base <- application_vgg16(
  weights = "imagenet",
  include_top = FALSE,
  input_shape = c(150, 150, 3)
)

model <- keras_model_sequential() %>% 
  conv_base %>% 
  layer_flatten() %>% 
  layer_dense(units = 256, activation = "relu") %>% 
  layer_dense(units = 1, activation = "sigmoid")

# stacked recurrent layers
model <- keras_model_sequential() %>% 
  layer_gru(units = 32, dropout = 0.1, recurrent_dropout = 0.5,
            return_sequences = TRUE, input_shape = list(NULL, dim(data)[[-1]])) %>% 
  layer_gru(units = 64, dropout = 0.1, recurrent_dropout = 0.5,
            activation = "relu") %>% 
  layer_dense(units = 1)

# bidirectional recurrent layers
model <- keras_model_sequential() %>% 
  bidirectional(
    layer_gru(units = 32), input_shape = list(NULL, dim(data)[[-1]])
  ) %>% 
  layer_dense(units = 1)

• tfruns package

• cloudml package

• tfdeploy package

• Successful deep learning requires a huge amount of experimentation

• This requires a systematic approach to conducting and tracking the results of experiements

The training_run() function is like the source() function, but it automatically tracks and records output and metadata for the execution of the script:

library(tfruns)
training_run("mnist_mlp.R")

# define flags and their defaults
FLAGS <- flags(
  flag_integer("dense_units1", 128),
  flag_numeric("dropout1", 0.4),
  flag_integer("dense_units2", 128),
  flag_numeric("dropout2", 0.3)
)

# use flag 
layer_dropout(rate = FLAGS$dropout1)

# train with flag
training_run("mnist_mlp.R", flags = list(dropout1 = 0.3))

Train on default CPU instance:

library(cloudml)
cloudml_train("mnist_mlp.R")

• Automatically uploads contents of working directory along with script

• Automatically installs all required R packages on CloudML servers

# Train on a GPU instance
cloudml_train("mnist_mlp.R", master_type = "standard_gpu")

# Train on an NVIDIA Tesla P100 GPU
cloudml_train("mnist_mlp.R", master_type = "standard_p100")

job_collect()

Attend Michael Quinn's Large scale machine learning using TensorFlow, BigQuery and CloudML Engine talk after the break (interop track) to learn more.

Content details will be announced at the talk!

• TensorFlow was built from the ground up to enable deployment using a low-latency C++ runtime.

• Deploying TensorFlow models requires no runtime R or Python code.

• Key enabler for this is the TensorFlow SavedModel:

  SavedModel provides a language-neutral format to save machine-learned models that is recoverable and hermetic. It enables higher-level systems and tools to produce, consume and transform TensorFlow models.

• TensorFlow models can be deployed to servers, embedded devices, mobile phones, and even to a web browser!

model <- keras_model_sequential( %>% )
  layer_dense(units = 256, activation = 'relu', input_shape = c(784),
              name = "image") %>%
  layer_dense(units = 128, activation = 'relu') %>%
  layer_dense(units = 10, activation = 'softmax',
              name = "prediction")

Note that we give the input and output layers names ("image" and "prediction")

# ...compile and fit model

# export model
library(tfdeploy)
export_savedmodel(model, "savedmodel")

library(cloudml)
cloudml_deploy("savedmodel", name = "keras_mnist")

Copying file://savedmodel/variables/variables.data-00000-of-00001 ...
Copying file://savedmodel/saved_model.pb ...
Copying file://savedmodel/variables/variables.index ...
/ [3/3 files][  1.9 MiB/  1.9 MiB] 100% Done                                    
Operation completed over 3 objects/1.9 MiB.

Model created and available in https://console.cloud.google.com/mlengine/models/keras_mnist

• Run on embedded devices (e.g. Raspberry Pi)

• Keras models can be converted to iOS CoreML

• Keras models can be deployed to the browser with Keras.js

• Additional projects:
  • TensorFlow Lite
  • deeplearn.js

Attend Javier Luraschi's Deploying TensorFlow models with tfdeploy talk after the break (interop track) to learn more!

• Straightforward to load already trained models into Shiny applications.

• Training code (and hardware!) not required for inference.

model <- load_model_hdf5("model.hdf5")    # Keras 
model %>% predict(input)

predict_savedmodel(input, "savedmodel")   # SavedModel

• One catch: data fed to deployed models must be preprocessed (e.g. normalized) in the same fashion as it was preprocessed for training.

• Ideally all preprocessing would happen within the deployed model (i.e. within the TensorFlow graph/runtime) so clients could pass raw data (e.g. text, images, time-series data, etc.)

• The tfestimators package has a solution for this (Feature Columns) however more work is still required for Keras.

• Clients can typically find a way to do the preprocessing but it's often awkward and inconvenient. We'd like to make it much more straightforward and automatic.

• Recommended reading

• Keras for R cheatsheet

• Gallery and examples

• Subscribe to the TensorFlow for R blog!

Slides: https://beta.rstudioconnect.com/ml-with-tensorflow-and-r/

TensorFlow for R: https://tensorflow.rstudio.com

Stay up to date at: https://tensorflow.rstudio.com/blog/

# Modify model object in place (note that it is not assigned back to)
model %>% compile(
  optimizer = 'rmsprop',
  loss = 'binary_crossentropy',
  metrics = c('accuracy')
)

• Keras models are directed acyclic graphs of layers whose state is updated during training.

• Keras layers can be shared by mutliple parts of a Keras model.